Amdahl code, redux #411

tkphd · 2022-06-23T21:26:53Z

This PR re-opens #407, which was merged prematurely, and relates to #409. Original summary and quoted comments follow.

This PR replaces the dummy hpc-intro-data.tar.gz with a tarballed version of ~~@ocaisa's~~ our amdahl program, hpc-intro-code.tar.gz, and substitutes that black-box program in place of the pi calculator. This is in keeping with our goal of reducing the "programming" aspect of hpc-intro to the bare minimum, focusing instead on the cluster and practicing job submission.

Note that, whereas the upstream amdahl package can be installed as a module, the version archived here is a stand-alone Python file named amdahl, with defaults of 30 seconds of work and a parallel proportion of 85%.

@reid-a and I are planning to teach this version on Friday, 24 June (~~two days hence~~ tomorrow).

@ocaisa wrote,

I'm a bit surprised by the tarball, in #378 you argued against going that route (for good reasons) so I wonder why to do that now?

@tkphd responded,

Excellent point @ocaisa. The primary answer is "expediency," as Andrew & I want to teach this tomorrow and do not have Modules on the cluster in question. (Its nascent replacement does!)

In #378 you checked in a tarball with a version number, suggesting a desire to iterate on the code. My view is that a tarball is for releases, not development code, which is why I suggested spinning it out into a separate repository.

We also have a tarball for Ep. 15, which the existing lesson uses to work through scp and tar, but the contents are never actually used. Substituting a new tarball allowed me to keep the bones of Ep. 15, but convert it to do something of actual use, i.e., downloading locally, uploading to the cluster, extracting, and running amdahl.

I also made changes to the amdahl[.py] script before tarballing. Some of this is useful and could go upstream, some not. Specifically,

The upstream amdahl.py can't be run stand-alone; I added an if __name__ == "__main__" clause to make it work.

I prepended a #!/usr/bin/env python3 and made it executable.

I dropped the .py extension to enhance the mystique and make it more of a black box.

@ocaisa commented,

The upstream package can be installed with pip3 install --user amdahl and already comes with the amdahl executable as part of that installation, but we can talk about it after your lesson

@tkphd asked,

@ocaisa is that something learners would each do? If so, that would mean explaining pip, which I sure do not want to do. Otherwise, amdahl works because pip sets an alias , or... ?

Edited to update Amdahl repo URL

…-amdahl-code" This reverts commit bdf8529, reversing changes made to 86229eb.

ocaisa · 2022-06-27T14:25:34Z

For me there are a few assumptions going on in general when delivering this course, one of those that you have knowledge/control over the environment of your users. The amdahl package is released in the way it is because it can work in a number of ways: if you have modules you can expose it that way (including encoding it's mpi4py dependency), if you don't you can still install it via pip (which also checks/installs the required dependency). Just shipping a tarball has a good chance of not working because there is the assumption that mpi4py is already available in the users environment.

Ultimately this sounds like something that belongs in a snippet as there a selection of available ways to make the code available which are all perfectly valid:

Make it available as a module
Install it via pip
Download the source itself and ensure you have the dep satisfied somehow before running it (may need some minor tweaking to work out-of-the-box as per @tkphd's previous comment above...but I would not drop the .py extension, instead change the command to execute the code)

tkphd · 2022-07-06T16:59:38Z

Thanks for the thoughtful feedback @ocaisa. My rationalization for the tarball is that Ep. 15 (transferring files) depends on a tarball, and previously, it was repackaged data from a Python lesson that is unrelated and is never used after that episode. This meant a lot of the transfer commands were boilerplate with no concrete filenames or paths, which is mystifying and seems counterproductive. Overhauling Ep. 15 with an amdahl tarball feels right, because it makes the concepts of transferring more concrete and gives them purpose.

You are right, of course, that mpi4py may be unavailable, and having the cluster admin install amdahl as a module or via pip is sensible. Can you update the README/docs in the amdahl repo to outline the recommended workflow for installation, and usage after it's installed?

I can revise the tarball in this PR to contain a driver script that will simply import amdahl and run it, so we can still work through the curl/wget, tar, and scp/rsync commands in Ep. 15, then run it in Ep. 16.

tkphd · 2022-09-01T12:00:58Z

In today's Coworking Hours, we had a vigorous debate about how to install amdahl. The best/obvious option is to python3 -m pip install amdahl, which will pull in mpi4py if it's not already installed. Possible breakage modes include

The login node is firewalled, and pip can't reach PyPI to get the code.
The PyPI version of mpi4py is incompatible with the installed MPI libraries.

Can folks with access try to install amdahl on their login nodes, and check whether it successfully runs?

python3 -m pip install amdahl
mpirun -np 4 amdahl

Thanks!

ocaisa · 2022-09-13T13:10:34Z

I read the notes of the coworking hour where this was discussed and I still think the easiest is to go with a pip install approach.

As mentioned in those notes, it is possible to do an offline pip install if you have the tarball available. You can get the latest version of the tarball with the URL https://github.com/hpc-carpentry/amdahl/tarball/main so there would be no need to maintain a versioned tarball in the lesson.
With a tarball on the system (via curl/wget, plus scp if needed), let's just assume that the $HOME directory is on a shared FS (usually pretty reasonable assumption). Installation would be:

python -m pip install --user <tarball>

This doesn't resolve any dependency problem you may have, but the most problematic one for our use case is MPI4PY. This can be installed with

env MPICC=/path/to/mpicc python -m pip install --user mpi4py

While I haven't tested it, I imagine that mpi4py also be replaced by a tarball in this command. The MPICC environment variable is only really needed if mpicc is not in your environment (or maybe you need a particular MPI C compiler).

tkphd · 2022-09-15T22:03:01Z

Following this month's meetings, I have a few things to change on this branch before it's ready to merge -- mostly, teaching pip as the installation method of choice with some caveats for mpi4py and offline installation instructions.

…-amdahl-code" This reverts commit bdf8529, reversing changes made to 86229eb.

tkphd · 2023-01-19T12:32:16Z

Using the Amdahl/MPI4PY tarballs to exercise "transferring files," make sure to cover the cluster configuration firewalls allowing users to "pull" from inside, "push" from outside, or both.

…l-code-redux

ocaisa

I think hpc-intro-code.tar.gz is still being shipped with this PR

_episodes/15-transferring-files.md

_episodes/16-parallel.md

_episodes/17-resources.md

tkphd · 2023-01-31T21:46:15Z

I think hpc-intro-code.tar.gz is still being shipped with this PR

Weird, I thought 8a6f91b deleted it. Taken care of in decb121, which also consolidates the lingering pi code in a new tarball.

Co-authored-by: ocaisa <[email protected]>

tkphd · 2023-02-06T22:23:08Z

Maybe tools like https://www.parallelpython.com/ , https://www.dask.org/ and https://github.com/pgiri/dispy should be mentioned? Not all people will need MPI, and these might provide good alternatives when task farming is the main purpose.

Out of scope for this PR. Please file a separate issue or, even better, a PR setting MPI in context with these other tools/approaches, bearing in mind that there may be controversy over whether task farming is HPC (vs. HTC).

tkphd · 2023-02-16T01:48:00Z

I believe the requested changes have been made. Please re-review.

ocaisa

LGTM

_episodes/16-parallel.md

ocaisa · 2023-02-16T14:57:37Z

_episodes/16-parallel.md

-Finally, we'll ensure the `my_pi` through `print` lines only run on rank 0.
-Otherwise, every parallel processor will print its local value,
-and the report will become hopelessly garbled:
+Then submit your job. Note that the submission command has not really changed


I'll create an issue for this later, but this is not necessarily true

ocaisa · 2023-02-16T15:16:37Z

Many many thanks @tkphd for putting so much effort into this PR!

Revert "Merge pull request #410 from carpentries-incubator/revert-407…

450fc54

…-amdahl-code" This reverts commit bdf8529, reversing changes made to 86229eb.

tkphd requested review from annajiat, mikerenfro, ocaisa, psteinb and reid-a as code owners June 23, 2022 21:26

tkphd marked this pull request as draft September 15, 2022 22:00

Revert "Merge pull request #410 from carpentries-incubator/revert-407…

11ad96c

…-amdahl-code" This reverts commit bdf8529, reversing changes made to 86229eb.

tkphd added 3 commits January 19, 2023 14:12

Substitute Amdahl tarball for lesson files

8a6f91b

Merge branch 'amdahl-code-redux' of github:tkphd/hpc-intro into amdah…

5403b6a

…l-code-redux

pip install amdahl in parallel episode

2309e99

tkphd marked this pull request as ready for review January 19, 2023 20:59

tkphd added 2 commits January 19, 2023 16:31

fix missing blockquote tags

971442d

see amdahl not cpi in history

64e034a

tkphd force-pushed the amdahl-code-redux branch 2 times, most recently from 144736e to 16ed34a Compare January 19, 2023 21:56

update Python version

0eb4195

tkphd force-pushed the amdahl-code-redux branch from 16ed34a to 0eb4195 Compare January 19, 2023 22:06

tkphd added 2 commits January 19, 2023 17:17

updated server name

fb93453

a few readability fixes

115f82d

ocaisa requested changes Jan 31, 2023

View reviewed changes

remove unused tarball, consolidate pi code in new tarball

decb121

Ask learners to think about why a transfer may not work

2155be8

Co-authored-by: ocaisa <[email protected]>

tkphd and others added 6 commits February 2, 2023 08:09

Ask learners to think about why a transfer may not work

de82b5d

Co-authored-by: ocaisa <[email protected]>

Expand on the remote_destination for scp

4181a65

Co-authored-by: ocaisa <[email protected]>

Clarify that remote_path may not exist

5eafc00

Co-authored-by: ocaisa <[email protected]>

remove unused tarball, consolidate pi code in new tarball

2851268

discuss the tarball before transferring it

fac84f2

Merge branch 'amdahl-bad-merge' into amdahl-code-redux

1d489ba

tkphd added 5 commits February 10, 2023 18:18

clean up & clarify a thing or two

7dfc411

revised discussion of tar and scp flags & fields

7b9c78e

use nano to edit bashrc

9b656ad

substitute previously-seen example job ID

470c4c5

resolve build failure

62fd20d

tkphd requested review from wirawan0, bkmgit and ocaisa and removed request for wirawan0 and bkmgit February 16, 2023 01:47

tkphd requested review from wirawan0 and bkmgit and removed request for bkmgit and wirawan0 February 16, 2023 12:05

ocaisa approved these changes Feb 16, 2023

View reviewed changes

Update _episodes/16-parallel.md

160e626

ocaisa merged commit 3117db8 into carpentries-incubator:gh-pages Feb 16, 2023

tkphd deleted the amdahl-code-redux branch February 16, 2023 21:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Amdahl code, redux #411

Amdahl code, redux #411

tkphd commented Jun 23, 2022 •

edited

Loading

ocaisa commented Jun 27, 2022

tkphd commented Jul 6, 2022

tkphd commented Sep 1, 2022 •

edited

Loading

ocaisa commented Sep 13, 2022

tkphd commented Sep 15, 2022

tkphd commented Jan 19, 2023

ocaisa left a comment

tkphd commented Jan 31, 2023

tkphd commented Feb 6, 2023

tkphd commented Feb 16, 2023

ocaisa left a comment

ocaisa Feb 16, 2023 •

edited

Loading

ocaisa commented Feb 16, 2023

Amdahl code, redux #411

Amdahl code, redux #411

Conversation

tkphd commented Jun 23, 2022 • edited Loading

ocaisa commented Jun 27, 2022

tkphd commented Jul 6, 2022

tkphd commented Sep 1, 2022 • edited Loading

ocaisa commented Sep 13, 2022

tkphd commented Sep 15, 2022

tkphd commented Jan 19, 2023

ocaisa left a comment

Choose a reason for hiding this comment

tkphd commented Jan 31, 2023

tkphd commented Feb 6, 2023

tkphd commented Feb 16, 2023

ocaisa left a comment

Choose a reason for hiding this comment

ocaisa Feb 16, 2023 • edited Loading

Choose a reason for hiding this comment

ocaisa commented Feb 16, 2023

tkphd commented Jun 23, 2022 •

edited

Loading

tkphd commented Sep 1, 2022 •

edited

Loading

ocaisa Feb 16, 2023 •

edited

Loading